NO-JIRA: Track the total kubelet metrics outage durations with autodl framework #30593

dgoodwin · 2025-12-10T14:06:13Z

This is being done to track if we get better or worse and compare to past releases, will be backporting.

Also stopped generating metric endpoint down intervals if they overlap with node reboots. This should allow for more accurate tracking of this total.

openshift-ci-robot · 2025-12-10T14:06:16Z

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: automatic mode

openshift-ci-robot · 2025-12-10T14:27:33Z

Scheduling required tests:
/test e2e-aws-csi
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-csi
/test e2e-gcp-ovn
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi

dgoodwin · 2025-12-10T20:15:21Z

/label acknowledge-critical-fixes-only
/verified by dgoodwin

openshift-ci-robot · 2025-12-10T20:15:32Z

@dgoodwin: This PR has been marked as verified by dgoodwin.

Details

In response to this:

/label acknowledge-critical-fixes-only
/verified by dgoodwin

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

dgoodwin · 2025-12-10T20:15:35Z

Couldn't repro in the PR but the files are there.

Only generate metrics down intervals if they do not overlap with node reboots or updates. Sum the total time we were in metrics endpoint down on any node with a new generic monitortest for this purpose. Also sum high cpu intervals. This will allow us to track if we're making things better with changes and compare to past releases.

dgoodwin · 2025-12-18T13:12:17Z

/retest

dgoodwin · 2025-12-18T15:21:34Z

/pipeline required

openshift-ci-robot · 2025-12-18T15:21:38Z

Scheduling required tests:
/test e2e-aws-csi
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-csi
/test e2e-gcp-ovn
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi

dgoodwin · 2025-12-19T14:53:53Z

https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/30593/pull-ci-openshift-origin-main-e2e-gcp-ovn/2001674236535508992/artifacts/e2e-gcp-ovn/openshift-e2e-test/artifacts/junit/interval-duration-sum_20251218-163442-autodl.json

Sample file.

openshift-ci-robot · 2026-01-06T15:08:41Z

@dgoodwin: This pull request explicitly references no jira issue.

Details

In response to this:

This is being done to track if we get better or worse and compare to past releases, will be backporting.

Also stopped generating metric endpoint down intervals if they overlap with node reboots. This should allow for more accurate tracking of this total.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

dgoodwin · 2026-01-06T15:09:23Z

/retest

openshift-trt · 2026-01-06T19:11:36Z

Risk analysis has seen new tests most likely introduced by this PR.
Please ensure that new tests meet guidelines for naming and stability.

New tests seen in this PR at sha: c3bdb3e

"[Monitor:interval-duration-sum][Jira:"Test Framework"] monitor test interval-duration-sum cleanup" [Total: 15, Pass: 15, Fail: 0, Flake: 0]
"[Monitor:interval-duration-sum][Jira:"Test Framework"] monitor test interval-duration-sum collection" [Total: 15, Pass: 15, Fail: 0, Flake: 0]
"[Monitor:interval-duration-sum][Jira:"Test Framework"] monitor test interval-duration-sum interval construction" [Total: 15, Pass: 15, Fail: 0, Flake: 0]
"[Monitor:interval-duration-sum][Jira:"Test Framework"] monitor test interval-duration-sum preparation" [Total: 15, Pass: 15, Fail: 0, Flake: 0]
"[Monitor:interval-duration-sum][Jira:"Test Framework"] monitor test interval-duration-sum setup" [Total: 15, Pass: 15, Fail: 0, Flake: 0]
"[Monitor:interval-duration-sum][Jira:"Test Framework"] monitor test interval-duration-sum test evaluation" [Total: 15, Pass: 15, Fail: 0, Flake: 0]
"[Monitor:interval-duration-sum][Jira:"Test Framework"] monitor test interval-duration-sum writing to storage" [Total: 15, Pass: 15, Fail: 0, Flake: 0]

neisw · 2026-01-07T12:56:15Z

pkg/monitortests/testframework/intervaldurationsum/monitortest.go

+		TableName: "interval_duration_sum",
+		Schema: map[string]dataloader.DataType{
+			"IntervalSource":       dataloader.DataTypeString,
+			"TotalDurationSeconds": dataloader.DataTypeFloat64,


Do we expect the seconds to ever be fractional? Not a huge issue but curious.

Int would definitely be cleaner. I'll update.

neisw · 2026-01-07T16:03:51Z

/lgtm

/hold
just to validate the artifact gets picked up in ci-data-loader-test

neisw · 2026-01-07T19:23:59Z

pkg/monitortests/testframework/intervaldurationsum/monitortest.go

+		TableName: "interval_duration_sum",
+		Schema: map[string]dataloader.DataType{
+			"IntervalSource":       dataloader.DataTypeString,
+			"TotalDurationSeconds": dataloader.DataTypeInt64,


Whelp, looks like it is DataTypeInteger

neisw · 2026-01-07T19:24:18Z

/lgtm cancel

dgoodwin · 2026-01-07T20:50:11Z

/override ci/prow/okd-scos-images

openshift-ci-robot · 2026-01-07T20:50:23Z

Scheduling required tests:
/test e2e-aws-csi
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-csi
/test e2e-gcp-ovn
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi

openshift-ci · 2026-01-07T20:50:24Z

@dgoodwin: Overrode contexts on behalf of dgoodwin: ci/prow/okd-scos-images

Details

In response to this:

/override ci/prow/okd-scos-images

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

neisw · 2026-01-07T21:35:10Z

/lgtm

openshift-ci · 2026-01-07T21:36:24Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dgoodwin, neisw

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [dgoodwin,neisw]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

neisw · 2026-01-08T00:23:39Z

/hold cancel

Verified the artifacts are imported properly

openshift-trt · 2026-01-08T01:12:29Z

Risk analysis has seen new tests most likely introduced by this PR.
Please ensure that new tests meet guidelines for naming and stability.

New tests seen in this PR at sha: 63129f5

"[Monitor:interval-duration-sum][Jira:"Test Framework"] monitor test interval-duration-sum cleanup" [Total: 12, Pass: 12, Fail: 0, Flake: 0]
"[Monitor:interval-duration-sum][Jira:"Test Framework"] monitor test interval-duration-sum collection" [Total: 12, Pass: 12, Fail: 0, Flake: 0]
"[Monitor:interval-duration-sum][Jira:"Test Framework"] monitor test interval-duration-sum interval construction" [Total: 12, Pass: 12, Fail: 0, Flake: 0]
"[Monitor:interval-duration-sum][Jira:"Test Framework"] monitor test interval-duration-sum preparation" [Total: 12, Pass: 12, Fail: 0, Flake: 0]
"[Monitor:interval-duration-sum][Jira:"Test Framework"] monitor test interval-duration-sum setup" [Total: 12, Pass: 12, Fail: 0, Flake: 0]
"[Monitor:interval-duration-sum][Jira:"Test Framework"] monitor test interval-duration-sum test evaluation" [Total: 12, Pass: 12, Fail: 0, Flake: 0]
"[Monitor:interval-duration-sum][Jira:"Test Framework"] monitor test interval-duration-sum writing to storage" [Total: 12, Pass: 12, Fail: 0, Flake: 0]

dgoodwin · 2026-01-08T12:29:03Z

/verified by bigquery data being present

openshift-ci-robot · 2026-01-08T12:29:15Z

@dgoodwin: This PR has been marked as verified by bigquery data being present.

Details

In response to this:

/verified by bigquery data being present

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci-robot · 2026-01-08T14:32:30Z

/retest-required

Remaining retests: 0 against base HEAD 0d04a81 and 2 for PR HEAD 63129f5 in total

openshift-ci · 2026-01-08T18:59:16Z

@dgoodwin: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

openshift-ci bot requested review from deads2k and p0lyn0mial December 10, 2025 14:06

openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Dec 10, 2025

openshift-ci bot added the acknowledge-critical-fixes-only Indicates if the issuer of the label is OK with the policy. label Dec 10, 2025

openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Dec 10, 2025

dgoodwin force-pushed the kubelet-metrics-total-outage branch from aa24d89 to c3bdb3e Compare December 16, 2025 19:25

dgoodwin changed the title ~~Track the total kubelet metrics outage durations with autodl framework~~ NO-JIRA: Track the total kubelet metrics outage durations with autodl framework Jan 6, 2026

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jan 6, 2026

neisw reviewed Jan 7, 2026

View reviewed changes

openshift-ci-robot removed the verified Signifies that the PR passed pre-merge verification criteria label Jan 7, 2026

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 7, 2026

openshift-ci bot assigned neisw Jan 7, 2026

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Jan 7, 2026

neisw reviewed Jan 7, 2026

View reviewed changes

openshift-ci bot removed the lgtm Indicates that a PR is ready to be merged. label Jan 7, 2026

Track interval duration sums as ints not floats

63129f5

dgoodwin force-pushed the kubelet-metrics-total-outage branch from 1d81129 to 63129f5 Compare January 7, 2026 20:14

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Jan 7, 2026

openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 8, 2026

openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Jan 8, 2026

openshift-merge-bot bot merged commit f07be90 into openshift:main Jan 8, 2026
20 checks passed

NO-JIRA: Track the total kubelet metrics outage durations with autodl framework #30593

NO-JIRA: Track the total kubelet metrics outage durations with autodl framework #30593

Conversation

dgoodwin commented Dec 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci-robot commented Dec 10, 2025

Uh oh!

openshift-ci-robot commented Dec 10, 2025

Uh oh!

dgoodwin commented Dec 10, 2025

Uh oh!

openshift-ci-robot commented Dec 10, 2025

Uh oh!

dgoodwin commented Dec 10, 2025

Uh oh!

dgoodwin commented Dec 18, 2025

Uh oh!

dgoodwin commented Dec 18, 2025

Uh oh!

openshift-ci-robot commented Dec 18, 2025

Uh oh!

dgoodwin commented Dec 19, 2025

Uh oh!

openshift-ci-robot commented Jan 6, 2026

Uh oh!

dgoodwin commented Jan 6, 2026

Uh oh!

openshift-trt bot commented Jan 6, 2026

Uh oh!

neisw Jan 7, 2026

Choose a reason for hiding this comment

Uh oh!

dgoodwin Jan 7, 2026

Choose a reason for hiding this comment

Uh oh!

neisw commented Jan 7, 2026

Uh oh!

neisw Jan 7, 2026

Choose a reason for hiding this comment

Uh oh!

neisw commented Jan 7, 2026

Uh oh!

dgoodwin commented Jan 7, 2026

Uh oh!

openshift-ci-robot commented Jan 7, 2026

Uh oh!

openshift-ci bot commented Jan 7, 2026

Uh oh!

neisw commented Jan 7, 2026

Uh oh!

openshift-ci bot commented Jan 7, 2026

Uh oh!

neisw commented Jan 8, 2026

Uh oh!

openshift-trt bot commented Jan 8, 2026

Uh oh!

dgoodwin commented Jan 8, 2026

Uh oh!

openshift-ci-robot commented Jan 8, 2026

Uh oh!

openshift-ci-robot commented Jan 8, 2026

Uh oh!

openshift-ci bot commented Jan 8, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

dgoodwin commented Dec 10, 2025 •

edited

Loading